1. Field of the Invention
The present invention generally relates to semiconductor neural networks, and more particularly, to coupling elements which couple with specific coupling strengths (synapse loads) internal data input lines to internal data output lines, and to improvements of a neural network driving method.
2. Description of Background Art
In recent years, a variety of circuits modeled on a neuron of a human being has been contrived. Among such neuron models, there is one called a Hopfield's model. This Hopfield's model will be briefly described below.
In FIG. 1, there is shown a schematic structure of a unit modeled on a neuron. A unit i comprises an input portion A for receiving signals from other units k, j, and the like, a converting portion B for converting applied input signals according to a certain rule, and an output portion C for outputting the conversion results. The input portion A has a weight (synapse load) W for each input unit which indicates a coupling strength between the units. Therefore, an input signal Sk from the unit k is loaded with a weight Wik at the input portion A before transmitted to the converting portion B. This weight Wik can take any of positive and negative values or 0.
The converting portion B make a total sum "net" of inputs S that have been loaded with the weights W (when generically termed, the weight is referred to as "W" hereinafter) undergo a predetermined function f for output. Output Si from the unit i at the time t is given as: ##EQU1## As the function f, a threshold function shown in FIG. 2A or a sigmoid function shown in FIG. 2B is often used.
The threshold function shown in FIG. 2A is a unit step function having characteristics that when the total sum "net (i)" of inputs becomes larger than a threshold value .theta., "1" is output, and when it does not reach the threshold value, "0" is output.
The sigmoid function shown in FIG. 2B is a non-linear monotonously increasing function and given by the following expression: EQU f=1/[1+exp(-net(i))].
The range of values of the sigmoid function is from 0 to 1. Therefore, as the total sum "net (i)" of inputs becomes smaller, the output approaches to "0", and as the total sum "net (i)" of inputs becomes larger, the output approaches to "1". When the total sum "net (i)" of inputs is "0", this sigmoid function outputs "0.5".
Another function obtained by adding a predetermined threshold value .theta. to the above-mentioned sigmoid function, as given by the following expression, may be employed. EQU f=1/[1+exp(-net(i)+.theta.)]
The unit model above is modeled on a vital cell which receives stimuli from other neurons and fires when a total sum of the stimuli exceeds a certain value. The Hopfield's model provides an operational model to a network configured of a plurality of such neuron units.
In the expressions above, when one neuron is initialized, state of all the remaining neuron units is determined in principal by applying the above-mentioned two dynamic equations to each neuron unit and solving them simultaneously. When the number of units increases, however, it is almost impossible to investigate and catch hold of state of one unit after another, and to program weights and bias values such that an optimal solution can be provided for a target problem. Therefore, Hopfield has introduced, in place of state of each unit, an energy function E as a quantity for representing entire characteristics of a neural net, which is defined as follows. ##EQU2## In the expression above, Ii is a self-bias value specific to the unit i. Hopfield has demonstrated that when the weight (synapse load) Wij has a symmetry shown as Wij=Wji, each unit changes its own state such that the above-mentioned energy function E always takes minimum values (more correctly, local minima), and proposed that this model be applied to programming of the weight Wij. A model according to the energy function E as described above is called a Hopfield's model. The expressions above are often restated for a discrete model as: ##EQU3##
In the expression above, n is a discrete time. Hopfield himself has demonstrated that the Hopfield's model above can work with good accuracy especially when the function f indicating input/output characteristics has a steep gradient (which is approximate to a unit step function in which most of the outputs take values close to either "0" or "1").
Neural networks have been configured according to this Hopfield's model in VLSI (Very Large Scale Integration) technology. An example of such a neural network is disclosed in "Computer", March, 1988, pp. 41 to 49, published by IEEE (Institute of Electrical and Electronics Engineers), or in "A CMOS Associative Memory Chip Based on Neural Network", by H. P. Graf in "87 ISSCC, Digest of Technical Papers", 1987 February, pp. 304 to 305, published by IEEE.
In FIG. 3, there is shown the entire schematic structure of a conventional integrated neural network circuit disclosed in the documents above. Referring to FIG. 3, the conventional integrated neural network circuit comprises a resistive matrix 100 having resistive coupling elements with predetermined weights arranged in a matrix, and an amplifying circuit 101 for amplifying potentials on internal data input lines included in the resistive matrix 100 and feeding back those amplified signals to the input portions of the respective resistive coupling elements. The resistive matrix 100 comprises the internal data input lines and internal data output lines arranged in a direction orthogonally intersecting the internal data input lines, as will be described in detail later. Interconnections between the internal data input lines and the internal data output lines made through the resistive coupling elements can be programmed by programming resistance values of the resistive coupling elements.
To select resistive coupling elements contained in the resistive matrix 100, there are provided a row decoder 102 and a bit decoder 103. The row decoder 102 selects one row of coupling elements in the resistive matrix 100. The bit decoder 103 selects one column of coupling elements in the resistive matrix 100.
To write coupling strength information in the resistive coupling elements selected by the row decoder 102 and the bit decoder 103, there are provided an input/output data register 104 for temporarily latching applied data, a multiplexer 105 for connecting the input/output data register 104, according to write/read mode of the data, to either the internal data input lines or the internal data output lines in the resistive matrix 100, an interface (I/O) 106 for connecting the input/output data register 104 to the outside of the device. This neural network is integrated on a semiconductor chip 200.
The row decoder 102 and the bit decoder 103 select a single resistive coupling element, to which information of coupling strength is written in through the input/output data register 104 and the multiplexer 105. Thus, states of the respective coupling elements contained in the resistive matrix 100 are determined, or interconnection states of the internal data input lines and the internal data output lines can be programmed in this manner.
FIG. 4 shows more specifically an example of structure of the resistive matrix 100 shown in FIG. 3. Referring to FIG. 4, the resistive matrix 100 comprises internal data input lines A1 to A4 and internal data output lines B1 and B1, B2 and B2, B3 and B3, and B4 and B4. At the connections between the internal data input lines A1 to A4 and the internal data output lines B1 and B1 to B4 and B4, there are provided resistive coupling elements 1. Each coupling element 1 can take three states; "open state", "excitatory state" and "inhibitory state". The state of each resistive coupling element 1 can be externally programmed according to an applied problem. Though in FIG. 4, those resistive coupling elements 1 that are in the open state are not shown, all the connections between the internal data input lines and the internal data output lines are provided with the resistive coupling elements 1. Each resistive coupling element 1 transmits, according to its own programmed state, signal potential level on the corresponding internal data output line Bi (Bi) onto the corresponding internal data input line Aj.
For the internal input lines A1 to A4, there are provided amplifying circuits C1 to C4 for amplifying signal potentials on the corresponding internal data input lines and transmitting the amplified potentials to the corresponding internal data output lines. Each of the amplifying circuits C1 to C4 has two inverting amplifiers 2a and 2b connected in series. The inverting amplifier 2a inverts potential on the input line Ai and transmits the inverted potential onto the internal data output line Bi. The inverting amplifier 2b transmits the signal potential on the input line Ai onto the complementary internal data output line Bi.
Each of the resistive coupling elements 1 couples output of one amplifier Ci to input of another amplifier Cj. A specific structure of the resistive coupling element 1 is shown in FIG. 5.
Referring to FIG. 5, the resistive coupling element 1 comprises current limiting resistive elements R+ and R-, random access memory cells 150 and 151 for storing coupling strength information, switching elements S1 and S2 responsive to an output signal of an amplifying circuit Ci to be turned on/off, and switching elements S3 and S4 responsive to the information stored in the random access memory cells 150 and 151 to be set in the on/off state. The resistive element R+ has one terminal connected to a supply potential V.sub.DD. The resistive element R- has one terminal connected to another supply potential (for example, ground potential) V.sub.SS. The switching element S1 is controlled by output of an inverting amplifier 2b for its on/off. The switching element S2 is turned on/off according to the information stored in the random access memory cell 150. The switching element S3 is set in the on/off state according to the information stored in the random access memory cell 151. The switching element S4 is controlled by output of another inverting amplifier 2a for its on/off.
In order to write information indicative of coupling strength into the random access memory cells 150 and 151, word lines WLP and WLQ and a bit line BL are provided. The random access memory cell 150 is provided at the crossing of the word line WLP and the bit line BL. The random access memory cell 151 is provided at the crossing of the word line WLQ and the bit line BL. The random access memory cell 150 stores information indicative of "excitatory state" and the random access memory cell 151 stores information indicative of the "inhibitory state". Thus, two word lines are provided for a single coupling element. The word lines WLP and WLQ receive row select signals from the row decoder 102 shown in FIG. 3. The bit line BL is selected by the bit decoder 103 shown in FIG. 2 to receive coupling strength information. The word lines WLP and WLQ are provided in parallel with the internal data input line Ai, and the bit line BL is in parallel with the internal data output line Bi in the resistive matrix.
In the structure shown in FIG. 5, output of an amplifying circuit Ci does not directly supply current to a corresponding internal data input line, thereby reducing output load capacitance of the amplifying circuit Ci. The coupling element 1 can selectively take three states, as described above, according to the programmed states of the random access memory cells 150 and 151. That is, the "excitatory state" where the switching element S2 is in on the state (active state), the "inhibitory state" where the switching element S3 is in the on state (active state), and the "open (don't care) state" where both switching elements S2 and S3 are in the off state (non-active state).
When potential levels on output lines Bi and Bi of an amplifying circuit Ci coincide with a programmed coupling state of a certain resistive coupling element 1, current flows into a corresponding data input line Ai either from the supply potential V.sub.DD or from the other supply potential (ground potential) V.sub.SS. When the programmed coupling state of the resistive coupling element 1 is "open, no current flows through the input line Ai irrespective of output state of the amplifying circuit Ci.
When the above-mentioned circuit model is compared to a neuron model, the amplifying circuit Ci corresponds to a neuron body (the converting portion B in FIG. 1). The signal lines A1 to A4, and B1 to B4 and B1 to B4 correspond to the data input portion A and the data output portion C (or dendrite and axon) shown in FIG. 1, respectively. The resistive coupling element 1 corresponds to a synapse loading portion which provides weighting between neurons. Now, operation of the resistive matrix will be briefly described.
The model shown in FIG. 4 is often called a connectionists' model. In this model, each neuron unit (amplifying circuit Ci) simply performs thresholding of an input signal, or outputs a signal corresponding to magnitude of the input signal compared with a predetermined threshold value. Each resistive coupling element couples output of one amplifying circuit Ci to input of another amplifying circuit Cj. Therefore, state of each amplifying circuit Ci is determined by states of all the remaining amplifying circuits Cj (i.noteq.j). When a certain amplifying circuit Ci detects current on the corresponding input line Ai (i=1 to 4), output of the amplifying circuit Ci at that time is given as: ##EQU4## In the expression above, Vin (j) and Vout (j) represent input and output voltages, respectively, of the amplifying circuit Cj connected to an internal data input line Aj, Ij represents current flowing through a single resistive coupling element, Wij represents conductance of a resistive coupling element which couples the amplifying circuit Ci connected to the internal data input line Ai to the amplifying circuit Cj connected to the internal data input line Aj. The output voltage Vout of the amplifying circuit Ci is determined by transfer characteristics of the amplifying circuit Ci itself. The voltage applied to the amplifying circuit Ci from the input line Ai is given by a total sum of currents flowing into the input line Ai. This input voltage is adjusted such that the total current flowing in this network becomes 0. In such state, the total energy of the neural network reaches local minima.
Each of the amplifying circuits Ci is constituted of, for example, a CMOS inverter. When the CMOS inverter has a high input impedance and a non-linear monotonously increasing threshold function as described above, the following relational expression can be obtained from the above-described condition that the total current becomes 0. ##EQU5## In the expression above, Iij represents current flowing through the resistors of a resistive coupling element controlled by output of the amplifying circuit Ci connected to the input line Ai. .DELTA.Vij is a potential difference at the resistive coupling element and given by: ##EQU6## Rij represents resistance of the resistive coupling element and is given by R+ or R-. Therefore, the voltage Vin (i) is a total sum of all the outputs of the amplifying circuits connected to the data input line Ai.
The above-mentioned operation is analogical computation. This analogical computation is performed at a time in parallel in the resistive matrix 100. However, both input data and output data are digital data. Subsequently, a practical computing operation will be described with reference to FIG. 4.
Input data is applied to the respective internal data input lines A1 to A4 through a register 104. The respective internal data input lines A1 to A4 are charged to potential levels corresponding to the input data and thus the neural network is initialized.
Output potentials of the amplifying circuits C1 to C4 change according to charging potentials applied to the data input lines A1 to A4. These potential changes of the respective amplifying circuits C1 to C4, or potential changes on the internal data output lines are fed back to the internal data input lines A1 to A4 through the corresponding resistive coupling elements. The potential levels, or current values fed back to the internal data input lines A1 to A4 are defined by the programmed states of the respective resistive coupling elements. More specifically, when a resistive coupling element has been programmed to be in the "excitatory state", current flows from the supply potential V.sub.DD to a data input line Ai. On the other hand, when the resistive coupling element has been programmed to be in the "inhibitory state", current flows from the supply potential V.sub.SS to the data input line Ai. Such operations proceed in parallel except for those resistive coupling elements that have been set in the open state. Thus, currents flowing into the data input line Ai are analogically added together, causing a potential change on the data input line Ai. When the changed potential on the data input line Ai goes over a threshold voltage of the corresponding amplifying circuit Ci, output potential of this amplifying circuit Ci changes. By repeating such operation, output potential of each amplifying circuit Ci changes to meet the above-mentioned condition that the total sum of currents becomes 0, until the network settles in a state satisfying the above-described expression of the stable state.
After this neural network has been stabilized, the output voltage of the amplifying circuit Ci is stored in a register (the input/output register 104 shown in FIG. 3) and then read out. A determination as to whether the network has been stabilized or not is made, depending on whether or not a predetermined time has passed since the data input or by directly comparing succeeding output data stored in the output register and detecting difference therebetween in terms of time. In the latter case, it is determined that the network has been stabilized when differences between the compared output data get smaller than a predetermined value, and then output data is provided.
This neural network outputs such output data as allowing energy of the neural network to settle in minimum values (or local minima). Thus, according to the programmed states of the resistive coupling elements, the resistive matrix 100 stores some patterns or data and can determine match/mismatch between input data and the stored pattern or data. Therefore, such a neural network can also serve as an associative memory circuit or a pattern discriminator.
A structure obtained by removing the feedback paths between the internal data output lines Bi and Bi and the internal data input lines Aj in the resistive matrix 100 shown in FIG. 4 has been known as a perceptron circuit of a single layer. This perceptron circuit can have a simplified learning algorithm, and when multi-layered, a flexible system can be implemented.
FIG. 6 shows a specific example of a possible structure of the coupling element shown in FIG. 5. In FIG. 6, there is shown structure of a coupling element Tij disposed at a location of i row and j column in the resistive matrix, or at the connection of an internal data input line Ai and an internal data output line Bj, and the parts equivalent or corresponding to those in the conceptual structure of the coupling element shown in FIG. 5 are denoted by the same reference numerals.
In FIG. 6, each of the switching elements S1 to S4 is constituted of an insulated gate field effect transistor (MIS transistor). The internal data input line is formed of complementary signal lines Ai and Ai. Data complementary to each other are transmitted from the input register (see FIG. 3) onto these paired complementary signal lines.
The bit line BL is constituted of a complementary bit line pair of BLj and BLj receiving complementary data.
The random access memory cell (RAM1) 150 comprises two inverting amplifiers IN1 and IN2 for storing coupling strength information that are anti-parallel to each other, or cross-coupled with the input portion of one amplifier connected to the output portion of the other, and MIS transistors TR1 and TR2 responsive to a signal potential on the word line WLiP for being turned on to connect the input portions of the inverting amplifiers IN1 and IN2 to the bit lines BLj and BLj, respectively. The inverting amplifiers IN1 and IN2 constitute a latch circuit which stores coupling strength information. The information latched by the latch circuit is transmitted through a node N1 to the control electrode (gate electrode) of the switching element (MIS transistor) S2.
Likewise, the random access memory cell (RAM2) 151 comprises inverting amplifiers IN3 and IN4 constituting a latch circuit, and MIS transistors TR3 and TR4 responsive to a signal potential on the word line WLiQ for being turned on to connect the input portions of the inverting amplifiers IN3 and IN4 to the bit lines BLj and BLj, respectively. The information stored in the latch circuit constituted of the inverting amplifiers IN3 and IN4 is applied through a node N2 to the control electrode (gate electrode) of the switching element(MIS transistor) S3. In the following, operation of writing the coupling strength information into this coupling element Tij will be briefly described.
When the word line WLiP is selected by the row decoder 102 (see FIG. 3), the MIS transistors TR1 and TR2 are turned on together, connecting the input portions of the inverting amplifier IN1 and IN2 to the bit lines BLj and BLj, respectively. Subsequently, data of "0" and "1" are transmitted onto the bit line BLj and its complementary bit line BLj, respectively. Then, due to the latching function of the cross-coupled or anti-parallel inverting amplifiers IN1 and IN2, data of "1" is stored at the node N1. Thus, the "excitatory state" of the coupling element Tij is programmed.
When this coupling element Tij is to be programmed in the "inhibitory state", the word line WLiQ is selected and data of "0" and "1" are transmitted onto the bit line BLj and the complementary bit line BLj, respectively. Due to the latching function of the cross-coupled inverting amplifiers IN3 and IN4, data of "1" is latched at the node N2. When the coupling element Tij is to be programmed to take the "open state", the word lines WLiP and WLiQ are sequentially selected and data of "1" and "0" are transmitted onto the bit line BLj and the complementary bit line BLj, respectively. As a result, data of "0" are latched at the nodes N1 and N2. Meanwhile, information of "1" represents an "H"-level signal and information of "0" represents an "L"-level signal.
While the coupling element shown in FIG. 6 is configured such that signal potentials on the internal data input lines are transmitted to the internal data output lines, the structure becomes equivalent to that of the coupling element shown in FIG. 5 if it is adapted such that signals on the internal data output lines Bj are fed back to the internal data input lines Ai (Ai).
FIG. 7 shows the entire structure of a neural network obtained by arranging the coupling elements as shown in FIG. 6 in a matrix of 4 rows and 4 columns.
In FIG. 7, in order to write coupling strength information into the coupling elements, there are provided a RAM I/O 106b serving as interface for transmitting and receiving data to and from outside of the device, selective gates 111 responsive to a column select signal (column decode signal) from the bit decoder 103 for connecting the selected column to internal data buses I/O and I/O, data registers 104 provided corresponding to the respective columns for amplifying and latching the applied data, and transfer gates 112 responsive to an operation mode indicating signal MUX for connecting the data registers 104 to the resistive matrix 100.
A column select line is formed of a complementary bit line pair of BL and BL and, therefore, the selective gates 111 and the transfer gates 112 comprise one pair of MIS transistors for each column. Among the MIS transistor pairs contained in the selective gates 111, one pair of MIS transistors are rendered conductive in response to the column select signal from the bit recorder 103. The transfer gates 112 are rendered conductive in a program mode where information of coupling strength is written in the respective coupling elements of the resistive matrix 100, and turned off in a practical processing operation where input data to be processed by the neural network is externally applied.
The RAM I/O 106b transmits a complementary data pair to the internal data buses I/O and I/O.
Word lines WL1P to WL4P and WL1Q to WL4Q are connected to the output portion of the row decoder 102 to receive a row select signal from the row decoder 102.
An input register 106a, which corresponds to the input/output register in FIG. 3 or the data input portion of an interface, has its output portion connected to complementary internal input data line pairs of A1 and A1 to A4 and A4 for transmitting complementary input signals (input data signals to be processed).
Amplifying circuits 101 are provided corresponding to the respective internal data output lines B1 to B4 to amplify signal potentials thereon.
In writing the information of coupling strength, the signal MUX attains the "H" level indicative of the active state so that the transfer gates 112 are rendered conductive. Subsequently, the row decoder 102 and the bit decoder 103 select one row and one column, respectively, and then desired information is written in the storage elements of the coupling element located at the connection of the selected row and column. At this time, the data register 104 latches the complementary data transmitted to the bit lines BL and BL and at the same time, the data are written in the coupling elements. Though each storage element has a latch circuit constituted of cross-coupled inverters, the data register 104 has a larger driving capability than the latching capability of the storage elements. Therefore, desired coupling strength information is written in the respective coupling elements.
In operation, the signal MUX falls to the "L" level indicative of the "inactive state", the transfer gates 112 are turned off. Subsequently, a neuron input signal, which has been applied from outside of the chip and temporarily stored in the input register 104, is transmitted to the internal data input lines Ai and Ai as an input signal to be processed in the resistive matrix 100. In the resistive matrix 100, according to the combination of the information stored in the RAM1 and RAM2 contained in the respective coupling elements Tij, charging and discharging operations are performed for the internal data output lines Bi in parallel. Voltage values on the internal data output lines Bi are detected and amplified by the amplifying circuits 101 and the results are developed as output signals.
In the structure above, a non-Hopfield's type neural network has been described where paths for feeding back signal voltages on the internal data signal lines to the resistive matrix are not provided. However, also a Hopfield's type neural network can be configured in the same manner if only paths for feeding back the internal data output lines Bi to the internal data input lines Ai and Ai are added to the above-described structure.
A conventional coupling element comprises storage element portions for storing coupling strength information and a current supplying element portion responsive to the information stored in the storage element portions and a signal potential on an internal data input line (or internal data output line) for transmitting a predetermined current to an internal data output line (or internal data input line). Therefore, as the number of elements constituting a coupling element increases with its structure getting more complicated, the area occupied by the coupling element becomes larger.
Further, since the word lines and the bit lines for writing coupling strength information in the storage element portions and the internal data input lines and the internal data output lines for transferring data to be processed are provided individually, a large number of signal lines are required, occupying a large area. Furthermore, since the large number of signal lines have to be provided in a small area, layout of the signal lines becomes complicated, resulting in a significant obstacle in achieving higher integration. Thus, the provision of a large number of signal lines, combined with complicity of the structure of coupling element, brings about disadvantages also in terms of production yield.
Additionally, in the conventional semiconductor neural networks, the input data signal continues to be applied for a significant time to sufficiently charge or discharge the internal data inputs lines to the "H" or "L" level, so that when the internal data input lines and the internal data output lines are charged or discharged in operation, potential of each signal line may make a full-swing. As a result, the consumption power becomes large and fast operability can not be achieved, while taking a longer processing time (or convergence time).
A structure of synapse load expressive unit is disclosed in "A Neuromorphic VLSI Learning System" by J. Alspector et al, pp. 323 to 325 in "Advanced Research in VLSI, 1987" published by MIT Press, where an input signal and an output signal of neurons are coupled together through a transistor which is turned on/off under control of a flip-flop storing a synapse coupling strength.